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APOLLO EXPERIENCE REPORT 


ACCEPTANCE CHECKOUT EQUIPMENT FOR THE APOLLO SPACECRAFT 

By I. J. Burtzlaff 
Manned Spacecraft Center 

SUMMARY 


The acceptance checkout equipment for spacecraft (ACE- SC) system is a com- 
puterized checkout system which provides for centralized, programed control of space- 
craft checkout operations in the Apollo Program. Most of the changes in the evolution 
of the ACE-SC system capability occurred in the computer programs. The initial de- 
sign of the ACE-SC was based on a concept of simple, computer -controlled, command- 
and-response checkout of the Apollo spacecraft systems. Because of the developmental 
modifications to the Apollo spacecraft, changes were required in the ACE-SC system to 
keep pace with the Apollo Program test requirements. 

The first major update was in the checkout- system software and was required to 
meet the test requirements for the guidance, navigation, and control system of the 
spacecraft. The second major change was implemented to optimize the use of ACE-SC 
memory. Because of the problems associated with accomplishing integrated testing of 
the Apollo spacecraft, approval was given in February 1968 for the addition of more 
memory capacity in the checkout system. Further system improvements included the 
capability for automatic data compression and automatic test sequencing during space- 
craft checkout. 

The ACE-SC system proved to be an effective tool; however, several undesirable 
conditions arose which should be corrected in the checkout systems and procedures used 
for future programs. These conditions can be summarized as follows. 

1. Total ground operations were not integrated within the ground checkout 
system. 

2. Different checkout systems were used for the booster and the spacecraft. 

3. Extensive use of special test equipment increased program costs. 


The unified-test- equipment concept, which has been proposed and is presently 
under development at the NASA Manned Spacecraft Center, significantly reduces these 
conditions in the following manner. 

1. General-purpose test equipment, rather than costly special test equipment 
that cannot be reused in subsequent test phases, is to be used during the vendor testing 
and the subsystem buildup. 

2. The test equipment is to be modular and expandable to meet the integrated- 
acceptance-test requirements during prelaunch operations. 

3. The total test- equipment complex is to be designed to use significantly less 
'ground-test equipment for an integrated test than was used in the Apollo Program. 

4. Upon completion of factory testing and integrated acceptance testing, the test 
equipment is to be used for operational spacecraft testing and the costly development 
of new test equipment thus avoided. 

5. The test equipment is to be designed to adapt easily to other NASA program 
checkout requirements. 

The experience gained with the ACE-SC used for the Apollo spacecraft is expected to 
lead to the development of checkout systems which will reduce significantly the cost of 
testing and checkout activities in future programs. 


INTRODUCTION 


The effectiveness of the testing and checkout phases of the Apollo missions has 
played a significant role in the accomplishment of a successful lunar- landing mission. 
The acceptance checkout equipment for spacecraft (ACE-SC) system has supported all 
Apollo spacecraft factory-acceptance testing, environmental- simulation testing, and 
prelaunch testing at the NASA John F. Kennedy Space Center (KSC). No major delays 
have been attributed to a failure of the ACE-SC system. Changes in testing and check- 
out requirements were important in stimulating the evolution of the ACE-SC system. 
Significant problems were involved in implementing major system changes and test- 
philosophy changes in the middle of a program as dynamic as the Apollo Program has 
been. This report provides a basic description of the ACE-SC system, a statement of 
the original goals and requirements for the ACE-SC system, a discussion of the 
ACE-SC system evolution and the problem areas encountered, and a definition of pro- 
posed checkout-system concepts that will resolve some of the undesirable conditions 
encountered in the use of the ACE-SC system during the Apollo Program. 

The information used in this report was accumulated through working experience 
with the ACE-SC system over a period of several years. The NASA and contractor 
personnel who have contributed to the success of the ACE-SC program (and hence to 
the information presented in this report) are acknowledged for the support and signifi- 
cant contributions they have provided. 
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DESCRIPTION OF THE ACE-SC SYSTEM 


The ACE-SC system is a computer- 
ized system that provides for centralized, 
programed control of spacecraft checkout 
operations. The ACE-SC system provides 
for manual, semiautomatic, and automatic 
operational modes to accommodate subsys- 
tem testing, integrated- system testing, and 
launch support. For the purpose of this 
discussion, the ACE-SC system (fig. 1) is 
considered in two parts, which are (1) the 
command subsystem and (2) the display- 
and-recording subsystem, also referred to 
as up-link and down- link subsystems, 
respectively. 


Figure 1.- Simplified data-flow diagram 
of the ACE-SC. 


Up-Link Subsystem 


The testing of spacecraft subsystems is controlled from selection-to-activate- 
random-testing (START) modules that are located in associated system consoles 
(fig. 2). Spacecraft subsystems that are tested by the various START modules include 
the environmental-control subsystem, the fuel-cell and cryogenics subsystem, the power 
and sequential subsystem, the guidance-and-navigation subsystem, the stabilization- and- 
control subsystem, the propulsion subsystem, the biomedical subsystem, the instrumen- 
tation subsystem, and the communications subsystem. The START modules facilitate 
the input of the appropriate command selections, computer- subroutine selections, or 
spacecraft-guidance-computer information to the spacecraft. 

Checkout- subsystem consoles can 

S t operate simultaneously with and independ- 

. a **»-'*® ently of other subsystem consoles. Each 

~ ~ console has a variety of test-command 

i ~ ~i -'u capabilities which are necessary for the 

• nr- r 1 testing and checkout of a particular space- 

(fig- 3) interprets and reacts to the com- 
|jja ;?r "' mands initiated from the subsystem con- 

‘TyWSfc JHPlC ' H i I sole. A specific command may instruct 

the computer either to initiate an automatic 
*. ' ' "" * test or to transmit a single command to the 

spacecraft. The signals generated by the 
ACE-SC ground station are transmitted by 
hardlines to the spacecraft vicinity, where 
the command signals are distributed by the 
digital test command system (DTCS). Re- 
dundant transmission checks and proper 
transmission verification tests are accom- 
plished to ensure maximum confidence in 
proper command transmission. 


Figure 2. - Typical ACE-SC control room 


DOWN-LINK SUBSYSTEM 



Figure 3. - Typical ACE-SC computer 
room. 


Test data to be processed by the down- 
link equipment are obtained from sensors 
in the spacecraft, from the carryon equip- 
ment, and from the ground- support equip- 
ment (GSE). The test data are transmitted 
as serial pulse-code-modulation (PCM) 
data to the recording and display equip- 
ment which receives, records, and dis- 
plays the spacecraft performance data, as 
required for the particular test procedure 
being conducted. The digital acquisition and 
decommutation equipment synchronizes on 
the incoming serial PCM bit stream, de- 
commutates the data, and routes the data 
for appropriate processing or display (or 
both). 


The down- link computer conducts the required processing, which includes such 
functions as predetermined limit checks, engineering unit conversions, data compres- 
sion, and a variety of special processing for each spacecraft subsystem. Display infor- 
mation is transferred to a symbol generator and storage unit, which generates 
alphanumeric -character display signals for display on the appropriate subsystem con- 
sole. The display can be specific parameters that will blink if the tolerance is 
exceeded, unique outputs based on special processing requirements, or status informa- 
tion for automatic test sequences. 


INITIAL APOLLO PROGRAM REQUIREMENTS FOR THE ACE-SC SYSTEM 


It was recognized early in the Apollo Program that the magnitude of the Apollo 
command and service module (CSM) and lunar module (LM) checkout requirements pre- 
cluded the use of manual test equipment because of the excessive checkout time that 
would have been required. The ACE-SC system underwent a significant evolution from 
the initial Apollo Program requirements to the final capability that was used in the 
Apollo 11 lunar- landing mission. A major part of ACE-SC system evolution took place 
in the ACE-SC computer-programing system, which had to be modified to meet the 
evolving test requirements as the missions became more complex. 

The initial requirements placed on the ACE-SC system for the early Apollo Pro- 
gram test phases were based on a concept of providing simple, computer- controlled 
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command-and- response checkout of the Apollo spacecraft. The capability to test sev- 
eral spacecraft subsystems simultaneously from one ACE-SC ground station also was 
included in the initial requirements. Basically, the requirements included the capa- 
bility to perform the following functions. 

1. To send a wide variety of test stimuli to the spacecraft from any ACE-SC 
subsystem console 

2. To monitor and display several hundred spacecraft parameters in real time 

3. To provide special processing and formatting of approximately 400 space- 
craft parameters for real-time display 

4. To provide complete documentation of the tests by recording all commands, 
recording digitally all out-of- limit parameters, recording raw PCM data, and record-- 
ing selected parameters on strip- chart recorders 


EVOLUTION OF THE ACE-SC SYSTEM 


Because of the evolutionary development of the Apollo spacecraft, changes were 
required in the ACE-SC system to keep pace with Apollo Program test requirements. 
The number of spacecraft parameters to be processed increased substantially through- 
out the program. Special computer programs had to be written to accomplish Apollo 
guidance- and- navigation- subsystem testing, to achieve automatic control for static fir- 
ing of the service -propulsion-subsystem engine, to control simulated altitude tests, to 
provide emergency detanking and spacecraft- system safing, and to meet many other 
requirements which caused the evolution of the ACE-SC hardware and software. The 
major advances which were accomplished during the ACE-SC system evolution are 
discussed in the following paragraphs. 


Software Design Changes 

Increased requirements . - The first major changes to the ACE-SC system capa- 
bility were accomplished by modification of the system software. These changes were 
necessary to meet the guidance-and- navigation- subsystem test requirements. The mod- 
ifications facilitated the interpretation by the ACE-SC system of guidance-computer digi- 
tal outputs, the loading of guidance-and-navigation tests from the ACE-SC system, and 
the recording of special guidance- computer data. In addition, the ACE-SC system soft- 
ware was expanded for additional spacecraft parameters and some special real-time 
data- processing functions. 

Memory limitations. - The next major system software change was made because 
of the ACE-SC system memory limitations. The addition of numerous parameters on 
the spacecraft, the addition of special processing requirements, and the addition of new 
GSE and launch- support requirements at the ESC caused the ACE-SC system memory 
capacity to be exceeded in 1966. The memory capacity of the ACE-SC system could 
not be increased at the time; therefore, a major effort was initiated to rewrite the sys- 
tem software, with the emphasis on conserving memory space. The system software 
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was altered to provide the capability to call alternate test loads, which caused seriali- 
zation of testing and constrained the amount of integrated testing that could be accom- 
plished. Control programs and subroutines were integrated, and common subroutines 
were used where possible to conserve computer memory. This method of resolving 
the immediate problem caused the computer processing capability to be exceeded when 
new programs were added to meet increasing test requirements. The trade-offs 
between memory capacity and computer processing time must be considered carefully 
for each checkout requirement to ensure that the optimum system software efficiency 
is obtained. Checkout systems for future programs must be designed for modular 
adaptation (of both hardware and software) to different and increasing testing and 
checkout requirements. 


Expansion of the ACE-SC System Memory 

As the testing and checkout activities evolved, it became apparent that additional 
memory capacity for the ACE-SC system would be required to provide the type of inte- 
grated testing necessary for the Apollo Program. Serialization of testing at the LM 
prime-contractor facility and at the KSC, as a result of alternate computer- program 
loads, would have had an intolerable impact on the schedule if the additional memory 
capacity had not been provided. The addition of 24 000 words of memory capacity was 
approved in February 1968, and modifications to the system software were made to 
allow use of the additional memory capacity. 


Data-Compression Techniques 

The amount of testing and checkout data that had to be processed for the Apollo 
Program became such an overwhelming problem that a requirement was generated by 
the Apollo Spacecraft Program Office to provide the capability to reduce significantly 
the amount of test data. This capability was provided in the ACE-SC system by initia- 
ting a decommutator design change which enabled the decommutator to perform fixed- 
limit checks and dynamic- limit checks (on a change greater than 1. 2 percent or on any 
data change) in conjunction with a computer program that filed only significant changes 
on digital tapes. In addition, this modification relieved the computer from some of its 
routine limit- checking functions and provided for the transmission of data to the com- 
puter in a direct-access mode. By providing for the transmission of data to the com- 
puter in a direct-access mode, a significant step was taken toward achieving the 
capability for automatic test sequencing during the testing and checkout of the Apollo 
spacecraft. 


Automatic Test Sequences 

One of the most significant steps in the evolution of the testing and checkout capa- 
bility for the Apollo Program was the implementation of a computer program, which 
was referred to as the Adaptive Intercommunications Routine (ADAP). The ADAP pro- 
vided the ACE-SC with the capability for closed- loop automatic test sequencing. "Adap- 
tive" refers to the capability of the program to adapt to new test requirements and to 
add new test sequences without affecting other computer-program functions. The term 
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” Intercommunications” refers to the closed- loop aspects of the programs, which pro- 
vide the capability to transmit up- link commands to the spacecraft, to receive the down- 
link response, and to initiate the appropriate action in a completely automatic mode. 

The status of the automatic operations is displayed in the ACE- SC control room, and 
appropriate override and restart- recycle capability is provided. Test sequences are 
called from digital tape, as required to meet the particular test- procedure requirements. 

The next generation of the ADAP system is being designed and will be imple- 
mented for Apollo telescope mount testing at the NASA Marshall Space Flight Center 
(MSFC). Of major significance is the capability of the ADAP program to generate test 
sequences in a higher level, engineering- oriented language to help bridge the communi- 
cations gap between engineering and programing personnel and to reduce the time re-* 
quired for definition of computer- programing requirements. 

Prior to the implementation of the ADAP program, there was no capability to 
provide special processing of checkout data without stopping the test or going to another 
available ACE-SC station. This problem has been reduced somewhat by using the ' 
ADAP test- sequence memory area and loading special data reduction programs to sup- 
port ” quick- look” requirements. 


Present Checkout- System Problems 

The ACE-SC system used on the Apollo missions does not adequately support all 
phases of ground testing that are desired in checkout systems for future programs. 
Vendor-acceptance and preinstallation testing are now delegated to special bench-test 
and acceptance- test equipment. The present ACE-SC system role begins after the sub- 
systems are installed in the spacecraft. A complete ACE-SC station (control-room and 
computer-complex combination) is configured for a CSM or an LM, regardless of the 
magnitude of testing required (that is, subsystem testing or integrated testing). A 
complete ACE-SC station is required for each vehicle (CSM and LM) during combined 
CSM and LM testing; consequently, redundancy occurs in the system hardware ele- 
ments. Station reconfiguration and site activation are lengthy processes that involve 
many test and operations personnel. 

Lack of adequate control and monitoring capability in spacecraft systems and 
GSE prevents the automation of the entire checkout operation by using the present 
ACE-SC system. The entire subsystem diagnostic capability is vested in the ACE-SC 
computer system, with no provisions for a built-in self-test in the spacecraft subsys- 
tems. The lack of provisions for a built-in self-test is a major constraint on the sim- 
plification of ground checkout systems. Future spacecraft subsystems should have 
built-in test logic to ensure adequate and efficient checkout operations. 


Digital-Test Command-System Problems 

One of the most significant problems encountered with the ACE-SC system was in 
the spacecraft-to-ACE-SC interface equipment. This equipment, which is referred to 
as the DTCS, is the means by which commands are decoded and routed to the appro- 
priate spacecraft test point. 
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On December 14, 1968, spacecraft 104 (the Apollo 11 command module) was in- 
advertently powered down as a result of the erroneous resetting of two latching relays 
within the DTCS located on Mobile Launcher no. 2. The cause of the problem was be- 
lieved to have been a transient voltage generated by some external source. There were 
subsequently 15 additional occurrences of the DTCS problem. 

A concentrated effort to solve the problem was initiated by the NASA Manned 
Spacecraft Center (MSC), the KSC, and all contractors that were associated with the 
use or manufacture of DTCS equipment. Because of the random nature of the problem, 
the source of the transients has never been definitely determined. 

Several significant points should be made for the purpose of improving GSE and 
facilities for future space programs. Improvements that need to be made can be sum- 
marized as follows. 

1. Susceptibility of critical GSE to transient voltage levels 

2. Complexity of GSE cabling and patching arrangements 

3. Adequacy of launch facility grounding systems, which require careful atten- 
tion during facility design and subsequent configuration control procedures that are 
comparable to those used for spacecraft systems 

An attempt to improve these areas in the early stages of future programs will not only 
reduce hazardous situations but should help reduce the time required to perform pre- 
launch operations. 


Mechanical and Radio-Frequency System Interface Problem 

One of the problems that impeded the advancement of automated checkout opera- 
tions was the lack of digital interfacing of mechanical, hydraulic, and radio-frequency 
(rf) test- support equipment with computerized checkout systems. Much of the time 
involved in checkout operations is consumed because voice communications are re- 
quired in order to coordinate the manual operation of valves, solenoids, and rf equip- 
ment. This equipment should be automated for future space programs. Automation 
will provide a reduction in the test time as well as in the number of personnel required 
to perform the more menial, repetitive operations. Automation of these areas will 
free technical personnel to apply their efforts to the analysis and evaluation of critical 
system areas and to the troubleshooting of system problem areas, which would provide 
for the more efficient use of manpower. 


Facility Problems 

Facility design and consideration of potential problems associated with power 
supply, equipment grounding, and equipment access can have considerable impact upon 
spacecraft testing if these areas are not given proper attention during the early stages 


of system development. Examples of problems which were encountered on the ACE- SC 
system during the Apollo Program include the following. 

1. The ACE-SC ground stations at the KSC underwent a significant number of 
power transients during spacecraft testing. In most cases, the ground- station com- 
puters required a program reinitialization or reload (or both) before the test activity 
could be continued. Power failures and transients also occurred at the contractor 
ACE-SC stations at Downey, California, and Bethpage, New York. After a total power 
failure at the prime-contractor site, a high incidence of ACE-SC hardware failures 
occurred. Normally, the failures were detected by the preoperational test programs 
and the diagnostic programs before spacecraft testing was continued. 

2. Air-conditioning deficiencies and failures created various ACE-SC system 
problems. At the contractor facility, a broken water main to the air-conditioning sys- 
tem caused the shutdown of all ACE-SC stations. After the air-conditioning system . 
had been repaired, the stations were powered up. Several ACE-SC system problems 
were detected and repaired before the spacecraft testing was continued. At the MSC, 

a deficiency in the air-conditioning system caused corrosion in the data- entry units in 
both ACE-SC control rooms. After several failures that were related to humidity and 
component corrosion had been identified, the affected elements were coated to reduce 
the effect of humidity on the system. The connections continued to corrode, however, 
and the problem was finally resolved by modifying the air-conditioning system to cor- 
rect the temperature and humidity problems. 

3. Isolated instances of poor equipment layout caused minor impact to spacecraft 
testing. The contractor recorded one incident in which two spacecraft being tested 
were interrupted as access to a unit in one control room had to be made through the 
control room of the other station. Another problem associated with poor equipment 
layout was the lack of isolated convenience outlets for the FR1400 recorders. Because 
the outlets were not isolated, there were instances of erroneous transients being re- 
corded with the test data. 

FUTURE CHECKOUT- SYSTEM OBJECTIVES 

To overcome the problems discussed 
previously, the MSC checkout personnel 
have established the following objectives 
for the next generation of testing and check- 
out equipment (fig. 4), which is presently in 
the development stages. 


1. To have a unified- test- equipment 
concept that will provide the standardiza- 
tion necessary to be capable of supporting 
subsystem, integrated, factory-acceptance, 
and launch-area testing (which would thus 
provide consistent test capability through 
all test phases and allow a complete data 
base to be established to support system 
operations) 



Present ACE-SC checkout 


Figure 4. - Test and checkout phases. 
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2. To have a modular subset of the standard system that will be capable of sup- 
porting preinstallation and bench- maintenance testing to reduce the requirement for 
special-purpose test equipment 

3. To have a built-in test capability both in subsystems and GSE to minimize 
ground diagnostic requirements 

4. To have universal control and display consoles that are designed to allow 
maximum automation, thus reducing the number of required test personnel 

5. To have a GSE control and monitor capability from the checkout station 

6. To have a basic system that will be capable of modular adaptation to the 
checkout requirements throughout the program evolution 

7. To have a central data facility for spacecraft system test history recording 
and data-processing support 

8. To have a serial digital interface with the spacecraft and GSE 


FUTURE CHECKOUT-SYSTEM CONCEPTS 


The next- generation checkout-system concept is described in the following sec- 
tion under the two basic categories which encompass the more significant features of 
the system: equipment configuration and system interfaces. A basic diagram of the 
system is shown in figure 5. 



Figure 5. - Space-station and shuttle- 
vehicle ground checkout station. 


Equipment Organization and Configuration 

The basic elements of the new system 
are the control and display console (fig. 6), 
the ground system interface units, the acqui- 
sition and control module, and a central 
computer facility. Multiple elements or 
portions of these elements will be used in 
the performance of all phases and levels of 
the testing and checkout activity, including 
unit, subsystem, and integrated- system 
tests. 


During checkout or prelaunch opera- 
tions, commands would be initiated from the 
control and display consoles and transmitted 
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Figure 6. - Control and display console 


to the spacecraft or the GSE. Responses would be received by the acquisition module 
and routed to the control and display modules for processing, display, and recording. 
Test history data would be recorded on the storage and retrieval module for subsequent 
use. The test history data would be transmitted to the central data facility on a non- 
interference basis. One console would be required for subsystem preinstallation tests. 


System Interfacing 

The system functional interfaces have three basic elements: (1) the space- station 
and space-shuttle hardware and the GSE, (2) the centralized computer facility, and 
(3) the operator interface. Each of these functional interfaces must be connected by a 
serial digital interface to provide the required information or data-acquisition 
capability. 

The complexity of the ground- system cabling presently used for the Apollo space- 
craft checkout will not satisfy future space-program requirements for simplicity, reli- 
ability, and fast turnaround time. The system under consideration would use serial 
digital up- link commands to control the spacecraft and PCM data for response monitor- 
ing. The interface between the checkout station and the GSE would be in the form of a 
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coaxial cable that carries information and data at a frequency of 1 to 5 megahertz. 
Because the central data facility would be performing a non- real-time function, stand- 
ard remote-terminal computer techniques would be used for test-area-to-central- 
facility communications. 

One of the major goals of the new system is to minimize the number of personnel 
involved in the on-line, real-time conduct of spacecraft tests, which would thereby min- 
imize the complexity of the man and machine interface. This goal can be accomplished 
by using the standard test console, by integrating the GSE with the checkout system, by 
providing specialist backup to the test conductors through standard television links, 
and by modular sizing of the control and display system to meet program phases. 


CONCLUSIONS AND RECOMMENDATIONS 


The ACE-SC system has performed all test and support functions in an outstand- 
ing manner. Major advances have been made in the automation of selected test se- 
quences, and confidence has been established in computerized systems for real-time 
checkout and evaluation of manned-spacecraft systems. However, major areas for 
improvement still remain that will contribute significantly to the efficiency of future 
testing and checkout programs. Areas for future improvements include the following. 

1. In the past, spacecraft systems and ground- support equipment have not 
always been designed with testing and checkout in mind. Future spacecraft and ground- 
support- equipment systems must have the designed- in capability for automatic 
checkout. 

2. The present checkout system does not provide the capability to support pre- 
delivery and preinstallation testing of spacecraft subsystems. Future checkout systems 
must be capable of modular adaptability to all phases of test activity as well as to 
changing test requirements. This will reduce the amount of special-purpose test equip- 
ment which has been required in the past as well as the modifications to the checkout 
equipment. 

3. The present ACE-SC system does not adequately provide for "quick -look" 
data processing with hard-copy data. Future checkout systems should be interfaced 
with a central computer facility to provide this capability as well as data -recording 
and trend -analysis capabilities. 

4. The communication of test requirements from engineers to the computer pro- 
gramer is a problem that can be reduced by the implementation of engineering-test- 
language compilers. Therefore, a test engineer with a minimum of training is allowed 
to submit requirements for test programs in engineering terms with which he is famil- 
iar. This procedure has been implemented with the Adaptive Intercommunication 
Routine, which is presently being used on the ACE-SC system; however, this computer 
program is highly oriented toward the ACE-SC system. A higher level test -language 
compiler is presently in development and will be made available for future testing and 
checkout activities. 
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The experience gained in the use of the ACE-SC system has been invaluable. It 
is anticipated that the benefit of this experience will contribute to the design of even 
more efficient and successful checkout systems for future programs. 


Manned Spacecraft Center 

National Aeronautics and Space Administration 
Houston, Texas, December 7, 1971 
914-50-17-08-72 


NASA-Langley, 1972 31 
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